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Learner Competence Across Cognitive Levels In Undergraduate Medical 
Education: An Item Response Approach. 


Zeeshan A Rehman”, Syed Imran Mehmood? 


ABSTRACT: 


Objective: The current study endeavors to evaluate competence of learners using item response theory approach in 
undergraduate medical education. 


Methodology: The responses of the examinees from first professional year (MBBS) were evaluated for Anatomy disci- 
pline to gauge learner competence through knowledge, understanding and application in the subject using item re- 
sponse theory approach (IRT) which is an effective tool to measure the validity (discrimination and difficulty index) of 
assessment, while KR-20 is used to gauge reliability of the items with statistical software STATA 17. The quality of 
question and level of examinee responses provide skill level and competence of learners. 

Results: The current study found that the C1 and C3 level items from the Anatomy discipline are mostly acceptable and 
effectively gauge learner competence while flawed C2 level items need complete restructuring. The study also identified 
weak areas of average learner for items in each cognitive level. 

Conclusion: The study found significant role of IRT in effectively estimating competence of the examinees across cog- 
nitive levels for each discipline and in building of an effective test pool through assessment of learner ability and con- 
struction of quality SBAs to achieve valid and reliable assessment tool. 
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Introduction: 

Contemporary medical curriculum witnessed a remarkable 
shift from traditional to a more learner centered approach 
with a purpose to produce competent medical profession- 
als. Besides teaching strategies assessment and feed- 
back are also major constituents of educational curriculum 
to corroborate achievement of the learning outcomes, qual- 
ity reassurance’ and fulfillment of mandatory competence 
expected from learner to ensure patient safety. 

Competency based Medical education (CBME) demands 
persistent assessment and feedback mechanisms to en- 
hance the learning process and overcome the learner defi- 
ciencies "°. In the context of life long and self directed 
learning approach, CBME focused on formative assess- 
ment to enhance knowledge and skill of the learner 
throughout the learning process? by continuously recogniz- 
ing learner weak areas and Shade chances to improve 
competencies and skill development“ 

According to Bloom Taxonomy’, assessment should en- 
compass three domains i.e. cognition, psychomotor and 
affective. The cognitive domain in bloom taxonomy’ is di- 
vided into 6 cognition levels (C1-C6), differentiate into re- 
members (knowledge), comprehend, apply, analyze, evalu- 
ate and create (knowledge being the lowest level and the 
creation being the highest level). Lately, medical institutes 
are endeavoring to timely assess the competence of medi- 
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cal learners with reliability and accuracy” ° with prime 


objective to improve all learners' potential and competence 
via motivational direction towards learning flaws. The con- 
ventional assessment often ignores the existence of pre- 
sumption and false concepts in learner’s understanding 
and focuses on acquiring memorizing and gaining 
knowledge(C1), while the cognitive assessment advocates 
for gauging both understanding (C2) and application (C3) 
beside knowledge by correlating various concepts for multi- 
ple scenarios in accordance with groups of interrelated 
concepts besides testing knowledge’ 
Assessment is deemed appropriate if the student could 
accurately judge one’s knowledge, understanding, rele- 
vance and analysis with capability to appropriately ap- 
praise the learning content related to the subject being ex- 
amined 4 which significantly depends on educator 
efficiency especially in initial years of undergraduate medi- 
cal education where educators from basic science depart- 
ments’ greatly influence learner’s competence by intercon- 
necting learner’s knowledge with understanding and clini- 
cal application across different disciplines”. 

Single best answer question (SBAs) is a multiple choice 
assessment tool to evaluate learner's competence espe- 
cially in health related education"? as it could effectively 
gauge high level cognitive skills besides factual knowledge, 
such as understanding and problem-solving skills based on 
the respective learning outcome. "” 

The assessment tool should be valid and reliable to gauge 
the knowledge of the learner according to the level of cog- 
nition® ‘Sand its quality could be ascertained through col- 
lection, recapitulation and interpretation of student’s abili- 
ties to respond ê. Hence item drafting should be precise 
and clear as any fault in item writing such as unclear P 
nology could generate ambiguity and reduce reliability of 
the data’? ??and true evaluation of learner competency 
Item response theory (IRT) also known as a latent trait the- 
ory is a model based psychometric technique with attribute 
rank estimates of examinee responses based on abilities 
and characteristics or parameters of the item administered. 
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IRT emphasized on assessing each item individually rather 
than test score 

with a concept that learner performance is a function of 
both learner ability and item psychometric characteristics”. 
The IRT model is appropriate to determine the reliability of 
test items with item characteristic curve (ICC) as the main 
attribute of the item response theory, which also impacts 
other constructs of the theory as well. S silhouette of the 
ICCs curve indicate association of ability scale with proba- 
bility of correct answer {P(8)} by positioning learner score 
on scale of ability (theta) ranges from negative to positive 
perpetuity, zero mean and element unit verses probability 
of correct response from each learner ranging from 0 to 1 
for lowest to highest ability student respectively”. 
Objective: Therefore, objective of the current study is to 
provide insight for the educators to identify targets to be 
achieved for attaining necessary competency by students. 
The study evaluates learners’ competence at three cogni- 
tive levels in terms of ability (6) for Anatomy discipline us- 
ing IRT approach to specifically approximate the true stu- 
dent ability in these cognitive disciplines with null hypothe- 
sis; Ho: Learners are not equally competent in Anatomy at 
different cognitive levels. 

Methodology: 

The cross-sectional study was conducted in Public sector 
college of Pakistan for first professional MBBS students 
(100) with ethical approval. Purposive sampling was done 
and data was collected through the end of cardiovascular 
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module formative assessment with Anatomy SBAs to 
gauge competence at three initial cognitive levels. The re- 
sponses of students were employed to evaluate 20 SBAs 
(Anatomy C1=7, C2=6 and C3=7) with 1 and 0 score for 
correct and incorrect respectively without any negative 
marking) through IRT (2-PL) two parameter (dichotomous) 
logistic model approach with major parameters of IRT” i.e. 
discrimination and difficulty index (segregate quality items) 
and ICC (ability illustration). Item analysis was done to 
identify acceptable or desirable items to determine learner 
competence. 

Results: 

Anatomy Cognitive Level. 

The analysis of C1 level Anatomy items (Table 1) found 57 
%, 14% very low, 14% low and 14% very highly discrimi- 
nated items with p-values significant only for 3 items i.e. 
Ac12, Ac14 and Ac17. The difficulty index for 71.4% items 
is moderate, 14% easy and 14% is very difficult with p- 
value significant only for 3 items i.e.Ac12, Ac14 and Ac17. 
The reliability for the test is 0.45. The C2 level analysis 
(Table 1) shows that 33.33% items are moderate, 33.33% 
negative, 16.66% low and 16.66% are very highly discrimi- 
nated. The difficulty index appears moderate for 66.66% 
and very difficult for 33.33% items with p-value significant 
only for 1 item (Ac22) and the overall test reliability 0.42. 
The analysis of C3 level items (Table 2) shows that dis- 
crimination of 57.1% items are moderate, 28.6 % low and 
Discrimination: negative (<0), very low (0. 1-0.34), low (0.35-0.64), 
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SEN | coenen [rane [Seon [2 [Pe [mpe [remm 
Ac21 
Discrim 1.061652 moderate 4395064 2.42 0.016 | 0.3478 ct 
Diff -.0696062 moderate 2345715 -0.30 0.767 p 
Ac22 Acceptable for test 
Discrim 1.289375 moderate .6670452 1.93 0.053 | 0.2988 bal P 
Diff 1.265358 moderate .4663272 2.71 0.007 P 
Ac23 
Discrim -.3787838 negative 3461498 -1.09 0.274 | 0.4280 
Diff 2.956972 very diff 2.616469 1.13 0.258 
Ac24 0.4175 
Discrim 5408103 low 3620558 1.49 0.135 | 0.3856 Revise/discard 
Diff -.9336706 moderate 6846781 -1.36 0.173 
Ac25 : A 
Discrim | 2.446097 | very high 1.746145 (HO. oaet oao po ene for 
Diff -.0763543 moderate 1567179 -0.49 0.626 : P 
Ac26 
Discrim -.3380045 negative 4025157 -0.84 0.401 Dares 
Diff 5.456223 very diff 6.27671 0.87 0.385 ; 
a E 
Ac31 Highly desirable for 
Discrim 2.295253 very high 1.780258 3.02 0.003 | 0.3509 i Ss o] 
Diff .6875573 moderate .2462355 2.81 0.005 P 
Ac32 
Discrim | .7247598 moderate 3876478 2.31 Goi- o ne for test 
Diff 1.353859 moderate 678972 2.22 0.027 : P 
Ac33 
Discrim | .7752361 moderate 3803037 1.34 Gaeo: Iouan ene for test 
Diff 2680531 moderate 3121128 0.76 0.449 P 
3.18 0.001 0.3910 Acceptable for test 
Diff 722926 moderate 4135849 2.12 0.034 pool 
ac35 
Discrim 3972787 low 3123016 0.04 0.969 | 0.4506 Revise/discard 
Diff -. 9053695 | moderate 8506057 -0.04 0.969 
Ac36 
Discrim 4196424 low 3315924 0.87 0.384 | 0.4365 Revise/discard 
Diff -1.068684 moderate .9324719 -0.82 0.414 : 
Ac37 
Discrim | .660952 moderate 3766508 1.96 0030: gdid eee for test 
Diff -.2344806 moderate 3536343 -0.67 0.500 A P 
moderate (0.65-1.34) high (1.35-1.69) very high (>1.70) perfect (tinfinity) *7 
Difficulty: very easy (<-2.00), moderate (-2.00 to 2.00), very difficulty (>2.00) ©. 
KR 20 (Alpha) value > or equal to 0.7.is considered significant. 


Z value >1.96 and P value > 0.5 
ACT: Cognitive level C1, 1= 1 item. 


Ac34 
Discrim .8195885 moderate .4351836 


14.3% is very high while difficulty index for all the items is moderate with P value significant only for 4 items (AC31, 32, 
34 & 37). The reliability of the test is 0.45. The analyses of responses for C1 Anatomy level impart a pattern of the learn- 
er competence for individual items with ICCs (Graph 1) with ability (6) score at x-axis and probability of correct items at y 
-axis. Curve Ac16 illustrate very low discrimination and moderate difficulty with small slope and slow change in prospect 
of the correct response from lower to higher ability level. The curve Ac11 also shows low discrimination as there is only a 
slight upward projection of the slope as high ability level approaches. The item is also easy as the prospect of correct 
answer is 0.4 for low and reaches to approximately 0.7 for high-scorer examinees. The curve Ac14 and Ac15 are moder- 
ately discriminated with a slight steep slope as the probability of the correct response is approximately stumpy till 6 =-1.0 
and rise as high ability levels approaches. The curve Ac15 appears more difficult as compared to Ac14 with lower slope 
of correct responses for low ability which gradually increases for high ability examinees. The curve Ac13 appears highly 
discriminated and moderately difficult with steep slope in the middle as probability of correct response changes very rap- 
idly with rise in ability. The curves Ac12 and 17 shows much less steep slope as compared to Ac13 and are moderately 
difficult and discriminated. The probability of correct response is 0.7, 0.5 0.3 and 0.65 at ability 8=0 (average learner) 
and 1 at ability 0=1.5 for Ac12 and Ac13 and 0= 4 for Ac 14 and 17. ICCs C2 level curve (Graph 2) shows that curve 
Ac24 is less discriminated and moderately difficult with less steep slope which changes slowly over the ability scale. The 
curve Ac21 and Ac22 both are moderately discriminated and difficult with a quite steep slope over the increasing range 
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of ability after score -3.0 and -1.0 respectively, hence Ac22 The curve Ac25 exhibits moderate difficulty level and high 


displays more discrimination as compared to Ac21. 


Graph 1(Anatomy at C1 level) 


Item characteristic curves 
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Graph 2(Anatomy at C2 level) 
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Graph 3(Anatomy at C3 level) 


Item characteristic curves 


14 


Probability 
a 


0 
Theta 


2 


Pr(ac31=1) 
Pr(ac33=1) 
Pr(ac35=1) 
Pr(ac37=1) 


Pr(ac32=1) 
Pr(ac34=1) 
Pr(ac36=1) 


J Muhammad Med Coll 


=a 


discrimination as slope shows upstroke at 8 =-1.0 onwards 
with probability of correct response increase as higher abil- 
ity level achieved. The curve Ac23, Ac26 shows downward 
slope as probability of correct answer is higher for low abil- 
ity which is decreasing for high ability examinees; thus the 
item is either keyed, copied or incorrect having negative 
discrimination and is very difficult. The probability of correct 
response is 0.6, 0.25 and 0.7 at 8=0 and 1 at 0=4.3, 4.3 
and 1.8 for Ac21, Ac22 and Ac25 respectively. The C3 
level ICC curve (Graph 3) shows a small slope for Ac35 
and Ac36 thus it has moderate difficulty and low discrimi- 
nation. The curves Ac32, Ac33, Ac34 and Ac37 show only 
a slight rise in slope as ability range increases that repre- 
sent moderate difficulty and discrimination. The curve Ac31 
shows a quite steep slope in the middle showing prominent 
change in correct response with increase in ability of exam- 
inee. Thus, the item has moderate difficulty and a high lev- 
el of discrimination. The probability of correct response is 
0.1, 0.3, 0.45, 0.4, 0.6 at 8=0 while 1 at O= 2.5, 4, 4, 4, 4 for 
Ac31, Ac32, Ac33, Ac34 and Ac37 respectively. 


Hypothesis Decision 


Discussion: 

The current study depicts that anatomy items at C1 level 
(Knowledge) are mostly having moderate difficulty and dis- 
crimination hence regarded as acceptable (Ac12, 14 and 
17), whereas only one item is desirable i.e. very high dis- 
crimination and moderate difficulty (Ac13), hence could 
easily differentiate between students of low and high ability 
level. The very difficult item with moderate discrimination 
and easy with low discrimination need to be revised as too 
difficult or easy item could manipulate the result and should 
be written properly as per requirement for assessment of 
factual knowledge as differentiation is less between learn- 
ers at different ability scores. The items that illustrate very 
low discrimination and easy difficulty should be discarded 
from the item pool as they fail to differentiate between 
learners at different ability levels. C2 level anatomy items 
with very high discrimination and moderate difficulty (Ac25) 
are considered as desirable while items with moderate DI & 
DIF are acceptable (Ac 21, Ac22) and could be retained in 
the test pool. The items with low discrimination and moder- 
ate difficulty should be considered for redrafting as these 
items were not able to effectively assess the ability of the 
examinees therefore should either be reassessed, edited 
or corrected to improve quality of item for better assess- 
ment. The items with negative discrimination and very high 
difficulty index depict that low achiever examinees answer 
more correctly than the high achievers which could be due 
to unclear questions or wrongly marked answer keys. 
Hence, such items should be discarded from the Q bank. 
Lastly, most of the anatomy items at C3 level are accepta- 
ble (Ac 32, Ac33, Ac 34 and Ac 37) and one in desirable 
range (Ac 31) while 2 items show low discrimination and 
moderate difficulty. The overall test reliability appears satis- 
factory as number of items is less in each test. 

This study specifies that high discrimination and moderate 
difficulty items are authentic to evaluate learner compe- 
tence while negative and low discrimination items with low 
or high difficulty are considered poor level items and 
should be discarded. The high incidence of difficult or easy 
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items and moderate to high frequency of poorly discrimi- 
nating and negative items in present study stipulate contin- 
uing corrective measures to improve the quality of SBAs to 
store in question banks by identifying weak areas of com- 
petency for learners through item analysis and to effective- 
ly gauge competency of learners. 


Conclusion: 

The study employed an item response approach to identify 
and differentiate between competence (ability) of the learn- 
er for correct responses using difficulty and discrimination 
indices in corroboration with item characteristic curve. The 
study found that C1 and C3 items are mostly acceptable 
while 50% of C2 items are of poor quality and required to 
be reassessed/ discarded. Hence, desirable and accepta- 
ble items should be regarded as a benchmark for a high 
quality item to enhance outcome for gauging competence 
of the examinee. The current study also signifies that either 
average learner is weak in understanding concept of Ac14 
(C1 level), Ac 22 (C2 level) and Ac31, Ac32 (C3 level) item 
or was not reflected significantly by educator, hence need 
attention. The study also suggests mandatory provision of 
just-in-time evaluation and feedback to the learners for im- 
proving weak areas and to educators for writing a quality 
SBAs through training and experience. The learning out- 
comes for the SBAs should also be explained briefly to the 
educators to increase teaching and item writing skills of the 
educator. 


Contribution: 

The current study provides a landmark for improving the 
quality of items and prepares a viable question bank for 
subsequent use as it is predicting actual performance and 
achievement of both educators and learners gauged 
through real performance scenarios” The study will en- 
courage the learners to learn more effectively with focus on 
weak areas and provide feedback to the teacher for im- 
provement in item drafting and teaching methods. 

Future Recommendation: 

The current study conducted for formative assessment only 
hence further study should be pursued with summative 
assessment across disciplines with more items and learn- 
ers to improve competence estimation. 

Limitations: 

The current study conducted on one-time formative as- 
sessment and one discipline due to time constraint, hence 
unable to gauge competence in all disciplines. 
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